Continuous Interaction with a Virtual Human

Attentive Speaking and Active Listening require that a Virtual Human be capable of simultaneous perception/interpretation and production of communicative behavior. A Virtual Human should be able to signal its attitude and attention while it is listening to its interaction partner, and be able to attend to its interaction partner while it is speaking – and modify its communicative behavior on-the-fly based on what it perceives from its partner. This report presents the results of a four week summer project that was part of eNTERFACE’10. The project resulted in progress on several aspects of continuous interaction such as scheduling and interrupting multimodal behavior, automatic classification of listener responses, generation of response eliciting behavior, and models for appropriate reactions to listener responses. A pilot user study was conducted with ten participants. In addition, the project yielded a number of deliverables that are released for public access

Modelling Paralinguistic Conversational Interaction Towards social awareness in spoken human-machine dialogue

Parallel with the orthographic streams of words in conversation are multiple layered epiphenomena, short in duration and with a communicative purpose. These paralinguistic events regulate the interaction flow via gaze, gestures and intonation. This thesis focus on how to compute, model, discover and analyze prosody and it’s applications for spoken dialog systems. Specifically it addresses automatic classification and analysis of conversational cues related to turn-taking, brief feedback, affective expressions, their crossrelationships as well as their cognitive and neurological basis. Techniques are proposed for instantaneous and suprasegmental parameterization of scalar and vector valued representations of fundamental frequency, but also intensity and voice quality. Examples are given for how to engineer supervisedlearned automata’s for off-line processing of conversational corpora as well as for incremental on-line processing with low-latency constraints suitable as detector modules in a responsive social interface. Specific attention is given t

Modelling Paralinguistic Conversational Interaction : Towards social awareness in spoken human-machine dialogue

Parallel with the orthographic streams of words in conversation are multiple layered epiphenomena, short in duration and with a communicativepurpose. These paralinguistic events regulate the interaction flow via gaze,gestures and intonation. This thesis focus on how to compute, model, discoverand analyze prosody and it’s applications for spoken dialog systems.Specifically it addresses automatic classification and analysis of conversationalcues related to turn-taking, brief feedback, affective expressions, their crossrelationshipsas well as their cognitive and neurological basis. Techniques areproposed for instantaneous and suprasegmental parameterization of scalarand vector valued representations of fundamental frequency, but also intensity and voice quality. Examples are given for how to engineer supervised learned automata’s for off-line processing of conversational corpora as well as for incremental on-line processing with low-latency constraints suitable as detector modules in a responsive social interface. Specific attention is given to the communicative functions of vocal feedback like "mhm", "okay" and "yeah, that’s right" as postulated by the theories of grounding, emotion and a survey on laymen opinions. The potential functions and their prosodic cues are investigated via automatic decoding, data-mining, exploratory visualization and descriptive measurements.QC 20120914</p